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Introduction 

THE CORNELL THEORY Center is one of four National Supercomputing Centers. During the 
ten years of its operation, the center has built a strong history in educational materials and 
workshops. Historically, the majority of teaching has taken place onsite at the Cornell Theory 
Center (CTC). In an age where electronic communications and access to information by 
computer is growing rapidly, it makes sense for us to turn our efforts to distance education on 
the Internet. 

We have designed and implemented a virtual workshop, a workshop where the participants are 
distributed across the United States (and occasionally further afield), learning by interacting 
with their own computers. In this paper we will describe our virtual workshop design and what 
we have learned from the first three hundred participants. The topics covered in our virtual 
workshop, by the nature of our mission, relate to technical, high-performance computing. We 
will not go into any detail about the contents of the workshop but describe instead the structure 
and implementation that we believe carry over to many other topics of education. The virtual 
workshop is a self-paced course. In addition to the on-line materials, we provide consulting 
support and access to our high-performance computer the IBM SP2. 

Goal: Offer to a More General Audience 

The virtual workshop was broadly advertised to both CTC users and the world at large. There 
was no charge for the workshop. Of the 290 applicants, 257 applicants were accepted, which was 
the largest number we felt we could support. 

There were two questions of particular interest in moving to a general audience. The first was 
whether more participants would encounter access problems, given that most were not already 
CTC account holders (who would be more likely to have already resolved access problems), and 
given that they were widespread geographically (in particular, internationally). 

The second was how active this audience would be, given that the course was self-paced and 
entirely voluntary. To answer this question, we collected information on use of modules, on 
participant involvement at the halfway point, and on CPU usage. CPU usage reflects how often 
they ran a program on our supercomputer. 

Audience Activity 
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The final evaluations asked participants to report which modules they had completed. The 
responses echoed the drop in activity with progression along the sequence of modules. Of the 89 
final evaluations, 60 reported completing the introductory MPI module. This number fell to 19 
and 9 for the last two of the 11 MPI modules. Participants with long-term userids, who might 
have the most immediate benefit from the course, progressed the farthest into the materials. 
They represented 30 percent of those who reported completing the first MPI module, but almost 
50 percent of those who reported completing the last two MPI modules. . 

Goals vs. Accomplishments 

To interpret the decrease in activity with progression into the modules, it is necessary to know 
whether the participants had any desire to complete those modules. 

In general a large proportion of the group (34 to 40, out of 44) planned to complete each 
module, indicating that they were interested in the later modules. For the early modules in a 
topic sequence, the number of participants who planned to and completed the module was very 
high compared with the number who planned to but did not complete the module. This "planned 
and completed" number drops off with progression into the topic sequence. 

CPU Usage 

In designing the course materials, we placed particular emphasis on lab exercises to be 
completed on the IBM SP2, in the belief that hands-on work is necessary to understand fully the 
course material. To determine how many of the participants worked on lab exercises, rather than 
just read Web materials, we examined their CPU usage at the end of the workshop. 

We could examine usage for only the 194 participants who did not have long-term CTC accounts. 
Of these, only 52 accumulated some CPU usage. If the same percentage of the 63 longer-term 
account holders worked on labs (probably an underestimate), the number doing lab exercises 
would be 69. 

Summary of Results from Offering to a more General Audience 

The sampling of activity from the Web server hits suggested that 74 percent of the participants 
accessed materials. The survey respondents who either had started, or planned to start, the 
virtual workshop, was 66 percent of the total participant group. These are rough indications of 
how many registrants participated to any extent in the course. The number participating per 
week was mostly likely much lower, with the Web server hits suggesting 23 percent of active 
participants. There was a decrease in activity with progression along the sequence of modules, 
despite preworkshop goals of completing those modules. 

To some extent, these trends were expected for a general audience and a self-paced course: 
education is often a discretionary activity displaced by higher-priority or time-limited activities. 
However, increased audience retention should be a goal for successive virtual workshops. Any 
improvement to the virtual workshop should have an impact on this measure, not just those 
specifically targeting audience retention. 

In general, participant activity or retention was not related to problems with Web access or SP 
access. 

C o n s u 1 tin g 
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In expectation of a heavy consulting load, a two-tier support structure was designed. There were 
nine front-line consultants (with varying amounts of time committed) who monitored incoming 
e-mail and answered questions within their range of expertise. The remaining questions were 
forwarded to the module developers. Three developers requested to answer all questions on their 
modules. 

Over the three months of the workshop 238 questions were received. Questions per week 
ranged from 5 to 32. For February, the month in which the most questions (and the most 
substantial questions) were received, 93 hours were spent on consulting by front-line 
consultants and by the three developers mentioned above. This level was well within the time 
originally allocated for consulting. Figures are not available for time spent consulting by other 
module developers. 

We were also interested in how many participants took advantage of consulting, and to what 
extent. We found that only 72 participants (out of 257 total participants, and 170 participants 
who were likely to have been active) used e-mail consulting. 

Goal: Revise and Expand Materials 

The trial workshop Web materials were revised, and new materials on MPI were added for this 
virtual workshop. It was in this area that the course received its most positive feedback. The 
mean overall ratings for the individual modules (compiled from the module evaluation forms) 
ranged from 3.67 to 4.33, where 1 = very poor and 5 = very good. The overall mean was 4.01. 

Goal: Impose Stricter Guidelines for Format of Web Materials 

For the trial workshop, all materials used a layered format, where one layer served as an outline 
into more detailed information. For this virtual workshop, we required that this layer also double 
as presentation foils. This was done to minimize staff effort for developing and maintaining 
materials. 

There were two approaches to layering the materials. One was to provide links wherever 
individual points needed expansion. So, one page might have none or many links. In the second 
approach, the presentation layer had one link per page, which accessed the corresponding 
material in a continuous discussion layer, For these materials, participants could read the 
presentation layer until they encountered material of particular interest, and then switch to the 
discussion layer; or they could simply read the discussion layer. The modules that received the 
highest overall ratings included modules using both of these approaches. 

In the final evaluation form, we obtained information on how the modules containing 
presentation and discussion layers were used, and information on participants' reactions to this 
format. Very few participants used only the presentation layer; the rest of the evaluations were 
split between using both layers and using just the discussion layer. The mean response to the 
question "Was this module organization helpful for you in navigating through the materials, or 
in focusing your study efforts on the sections of particular interest to you?" was 2.01, where 1 = 
very helpful and 5 = not at all helpful. The mean response improved with increased prior 
experience (from none to advanced): 2.50, 2.00, 1.72, 1.71. 

Given the positive response to the layered module structure, we will continue to require this for 
new virtual workshop modules. We believe that different topics might be better suited to 
different approaches to layering, and therefore we do not intend to standardize either of the 
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formats described above. The improved mean response from higher experience levels suggests 
that the presentation layer is being used to skim over familiar materials. 

Goal: Experiment with the Use of a MOO 

A MOO is a text-based, object-oriented, multi-user environment within which any number of 
people can interact, both with each other and with the environment. Activity is organized into 
"rooms" in which participants can speak to other occupants. Objects, such as bulletin boards, 
notes, and recorders, provide additional structure or functionality. 

The VWMOO was still in development when the course started, and was not available until the 
last third of the workshop. 

The original rationale for establishing a MOO was to provide a forum for interparticipant 
communication, and also for concurrent communication between participants and instructors 
(as either an alternative or supplement to e-mail consulting). A semi-automated training session 
on the VWMOO, informal office hours covering course materials, and group discussions on the 
virtual workshop format and distance education were offered. 

The VWMOO received very little activity. Only to of the 89 respondents to the final evaluation 
indicated that they had used the VWMOO. Self-motivated interparticipant communication did 
not occur, and office hours were poorly attended (maximum attendance was five participants). 
Most successful was the use of the VWMOO for informal, extended consulting. Reasons cited for 
not using the VWMOO were its late availability, insufficient notification of classes, and the 
primitive interface. 

The VWMOO will be available from the start of the next virtual workshop, notification of classes 
and office hours will be improved, and we hope to move eventually to a Web interface. Broader 
issues include how to integrate the VWMOO into the course materials and how to provide more 
structure to VWMOO activities. 

Overall Effectiveness 

There were three questions on the final evaluation form that aimed if to quantify the overall 
effectiveness of the virtual workshop format: 

How well do you believe your parallel programming learning goals have been met by 
participating in this workshop? 

Would you say you learned more or less from the virtual workshop than you would expect to 
learn from a face-to-face workshop? 

Would you say you learned more or less from the virtual workshop than you would expect to 
learn from an introductory textbook with exercises (but outside a class)? 

Parallel Programming Goals 

On a scale from I to 5, where 1 = completely and 5 = not at all, the mean rating for how well 
participants' parallel programming goals were met was 2.43. Participants with long-term CTC 
accounts had the most positive mean response: 2.13, as compared with 2.58 for corporate 
participants and 2.38 for academic participants. This could relate to the greater progress into the 
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materials made by the CTC users. Advanced parallel programmers had the most positive mean 
response: 1.64, as compared with 2.60, 2.37, and 2.83 for the three lower experience levels. 

On the final evaluation form, the participants were asked for "additional comments or 
explanations you have regarding your learning goals and the effectiveness of the workshop in 
helping you to achieve them." 

Both external and internal factors were cited. Broadly categorized, the external factors were 
insufficient time to commit to the workshop (21 respondents), shifts in job priorities and 
associated learning goals (3), instability in the home system (2), and slow network response (1). 

VW vs. Face-to-Face Workshop 

For the comparison with face-to-face workshops, on a scale of 1 to 5 where 1 = learned much 
more and 5 = learned much less, the mean response for all participants was 3.09. There was no 
marked variation with group or experience level. 

VW vs. Textbook 

For the comparison with a textbook, on a scale of I to 5, where 1 learned much more and 5 = 
learned much less, the mean response for all participants was 2.32. There was no marked 
variation with group, but the intermediate experience levels returned better ratings (2.15 and 
2.11) than either the lowest or highest experience level (2.55 and 2.70). 

Summary of Overall Effectiveness 

The "goals met" rating of 2.43 indicates that participants did benefit from the virtual workshop, 
but did not achieve all their goals. It is reassuring to note that external factors were a significant 
cause, that many participants did express satisfaction as to having their goals met, and that the 
reaction to the individual modules was very positive. 

We would have been (obviously) more pleased if the comparison with face-to-face workshops 
had been tipped in favor of the virtual workshop, since it aims to provide the same level of 
support as a live workshop, with the added benefit of no travel. 

Since the Web materials were very well received, the key to improving this rating might be to 
encourage use of all course resources. As discussed earlier, a substantial number of participants 
did not use the SP, did not submit consulting questions, and did not use the VWMOO. 

Another approach would be to add small interactive exercises throughout the modules; this 
could be anything from quizzes to short labs to Java-based demonstrations. 

Conclusions 

The winter 1995 virtual workshop succeeded very strongly in achieving three goals: scaling the 
workshop, adding material, and designing guidelines for the structure of materials. Although one 
goal, experimenting with the use of a MOO, was not fully implemented for this virtual workshop, 
our limited experience has provided valuable insight into how this tool can be used. Active 
participants were generally satisfied with the course, based on the module and final evaluations. 

This was CTC's first experience with self-paced, over-the-network training offered to a general 
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audience. We gained a realistic understanding of participants' commitment levels, and the 
factors that influence this. We intend to retain the self-paced format, as we believe this meets the 
needs of the widest possible audience, and is a good match to the over-the network approach in 
general, and to the use of Web-based materials in particular. We expect that improvements in 
format and materials can increase audience retention and activity. 

We will fine-tune the VW format. In particular, we will change our communication patterns with 
participants. For the next virtual workshop, we will ask participants to complete a biweekly 
survey. This will provide us with information needed to schedule activities and resolve problems; 
it will provide the participants with an opportunity to assess their progress, and will remind 
them of available services. In addition, we will provide the participants with "tips" at the 
beginning of the workshop, which will stress the importance of lab exercises, and we will more 
strongly advertise support activities. 

Among the other improvements and additions, we plan to add multiple choice quizzes that will 
allow participants to assess how well they have learned the material. We will also be 
experimenting with a Webbased chat facility in addition to the VWMOO. 

The virtual workshop is a flexible, accessible, and convenient educational tool for programmers 
and researchers of all experience levels. Future experimentation with format, presentation, and 
tools is needed to determine how to best meet individual needs, and how to strike a balance 
between motivating remote participants and preserving their independence. 
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